{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Learning - implementing elements of neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Last revision: Fri Jul  5 18:34:09 AEST 2019_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we will expand on some of the concepts of neural learning, starting with the perceptron. Initially we understand the representational capacity of a perceptron, then how to implement learning for elementary Boolean functions, i.e., concept learning, and look at a perceptron learning a linear classifier on a real-world dataset.\n",
    "\n",
    "The remainder of the lab goes into some \"hands-on\" aspects of supervised learning for neural networks, based on the multi-layer perceptron trained by error back-propagation. \n",
    "There are only questions as such in the first section, a review of perceptrons. For the second part on the multi-layer perceprton you are just supposed to step through the cells, running the code, understanding why it is doing what it does, and possibly adding your own cells to experiment.\n",
    "\n",
    "This code is for explanatory purposes only – for real neural networks you would use one of the many code libraries that exist. \n",
    "\n",
    "**Note: this notebook has only been tested using Python 3.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The perceptron implementation for this lab is based on the presentation and code in Chapter 3 of \"Machine Learning\" by Stephen Marsland, CRC Press, 2015. \n",
    "\n",
    "The multi-layer perceptron part of the lab is based on the presentation and code accompanying Chapter 18 of \"Data Science from Scratch\" by Joel Grus, O'Reilly Media, 2015 (all the code for the book is available [here](http://github.com/joelgrus/data-science-from-scratch))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (1) Linear classification with the Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we will use a slight variant on the notation and setup\n",
    "used in the lectures.\n",
    "These changes are not going to affect the capabilities of the perceptron.\n",
    "\n",
    "For a given set of $m$ inputs, the first stage of the computation is when the perceptron  multiplies each of the input values with its corresponding weight and adds these together:\n",
    "\n",
    "$$ h = \\sum_{i}^{m} w_{i} x_{i} $$\n",
    "\n",
    "The second stage is to apply the thresholding output rule or activation function of the perceptron to produce the classification output.\n",
    "\n",
    "For this lab we will slightly change the activation function to map to either $0$ or $1$ rather than the $-1$ or $+1$ we had in the lecture notes.\n",
    "\n",
    "The value set for the bias or threshold input will also be changed from $1$ to $-1$.\n",
    "\n",
    "$$ o = g(h) = \\left\\{\n",
    "                \\begin{array}{lll}\n",
    "                        1 & \\mbox{if}           & h > 0 \\\\\n",
    "                        0 & \\mbox{otherwise if} & h \\leq 0 \\\\\n",
    "                \\end{array}\n",
    "              \\right. $$\n",
    "\n",
    "Let's go ahead and implement a Perceptron in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing simple Boolean functions as a linear classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first look at modelling a simple two-input Boolean function as linear classifier. This is a Perceptron WITHOUT any learning! To get started we will use the OR function, for which the truth table will be familiar to you all. Note that you will need to pick some weights for the function to output the correct values given the input. There are many possible values that could do the job. Also, remember to take care with the dimension of the weight vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For Input [0, 0] with Class 0 Predict  0\n",
      "For Input [0, 1] with Class 1 Predict  1\n",
      "For Input [1, 0] with Class 1 Predict  1\n",
      "For Input [1, 1] with Class 1 Predict  1\n"
     ]
    }
   ],
   "source": [
    "# set up the data, i.e., all the cases in the truth table \n",
    "x=[[0,0],[0,1],[1,0],[1,1]]\n",
    "y=[0,1,1,1]\n",
    "# number of data points\n",
    "n=4\n",
    "# number of inputs to the perceptron\n",
    "m=3\n",
    "# what weights should be assigned to correctly compute the OR function ?\n",
    "# fill in your weights here by assigning a weight vector to w\n",
    "# e.g., w=[-0.09,0.14,0.01]\n",
    "w=[0.09,0.14,0.1]\n",
    "# how can you come up with a suitable set of weights ?\n",
    "# loop over the data\n",
    "for i in range(n):\n",
    "    h=w[0]*(-1)# this is the bias weight and input\n",
    "    for j in range(1,m):\n",
    "        h+=w[j]*x[i][j-1]\n",
    "    if(h>0):\n",
    "        output=1\n",
    "    else:\n",
    "        output=0\n",
    "    print('For Input', x[i], 'with Class', y[i], 'Predict ', output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now change your code to model the AND function (again restricted to two inputs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For Input [0, 0] with Class 0 Predict  0\n",
      "For Input [0, 1] with Class 0 Predict  0\n",
      "For Input [1, 0] with Class 0 Predict  0\n",
      "For Input [1, 1] with Class 1 Predict  1\n"
     ]
    }
   ],
   "source": [
    "# set up the data, i.e., all the cases in the truth table \n",
    "x=[[0,0],[0,1],[1,0],[1,1]]\n",
    "y=[0,0,0,1]\n",
    "# number of data points\n",
    "n=4\n",
    "# number of inputs to the perceptron\n",
    "m=3\n",
    "# what weights should be assigned to correctly compute the AND function ?\n",
    "# fill in your weights here by assigning a weight vector to w\n",
    "# how can you come up with a suitable set of weights ?\n",
    "# loop over the data\n",
    "w=[0.1,0.04,0.07]\n",
    "for i in range(n):\n",
    "    h=w[0]*(-1)# this is the bias weight and input\n",
    "    for j in range(1,m):\n",
    "        h+=w[j]*x[i][j-1]\n",
    "    if(h>0):\n",
    "        output=1\n",
    "    else:\n",
    "        output=0\n",
    "    print('For Input', x[i], 'with Class', y[i], 'Predict ', output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Changing the data structures for machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We got right down to the details of how a linear classifier works. Now this being a perceptron, you probably recall that rather than using a fixed set of weights to do the prediction each time, there is a simple training rule that updates the weights on the basis of discrepancies between the classifier's prediction on the data and the actual class. So we could extend our previous code to implement that training rule, but the code is a little fiddly and you're probably thinking there should be a simpler way to do this. If so, you are correct, but it is based on moving towards coding with matrix and vector operations, rather than directly using Python arrays. To do this we need to import the NumPy library (there is a tutorial at: <href <a>https://docs.scipy.org/doc/numpy-dev/user/quickstart.html</a>>)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, when we need to predict a class for an instance $\\mathbf{x}$ given the current weights $\\mathbf{w}$ we can use the inner product operation $\\mathbf{x} \\cdot \\mathbf{w}$. To get this functionality using NumPy we just do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.06\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "x=np.array([0,1,1])\n",
    "w=np.array([0.02,0.03,0.03])\n",
    "\n",
    "h=np.dot(x,w)\n",
    "print(h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But wait, there's more! Since $\\mathbf{x}$ and $\\mathbf{w}$ are both actually matrices, the same operation will enable us to apply the inner product of the weight vector $\\mathbf{w}$ to ALL the data instances at once. In this case we write the matrix of data instances $\\mathbf{X}$. Just note that we need to take care that the data matrix and weight vector are properly initialised to make this operation work correctly. Now the code for predicting the class values of all of our data given the weight vector is as follows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Activations:\n",
      " [[-0.02]\n",
      " [ 0.01]\n",
      " [ 0.01]\n",
      " [ 0.04]]\n",
      "Predictions:\n",
      " [[0]\n",
      " [1]\n",
      " [1]\n",
      " [1]]\n",
      "Misclassifications\n",
      " [[0]\n",
      " [0]\n",
      " [0]\n",
      " [0]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Data set with class values in last column\n",
    "dataset = np.array([[0,0,0],[0,1,1],[1,0,1],[1,1,1]]) # OR function\n",
    "X=dataset[:,0:2]\n",
    "y = dataset[:,2:]\n",
    "\n",
    "# Note: the bias weight is now the last!\n",
    "w = np.array([[0.03],[0.03],[0.02]])\n",
    "# Add the values for the bias weights (-1) to the data matrix\n",
    "nData = np.shape(X)[0]\n",
    "X = np.concatenate((X,-np.ones((nData,1))),axis=1)\n",
    "# get the value of the activation function\n",
    "h = np.dot(X,w)\n",
    "yhat = np.where(h>0,1,0)\n",
    "err = yhat-y\n",
    "\n",
    "print('Activations:\\n', h)\n",
    "print('Predictions:\\n', yhat)\n",
    "print('Misclassifications\\n', err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code uses some more NumPy built-ins. Check the documentation to be sure you know what is going on. One of these, np.where(), is useful here. It takes 3 arguments and returns an array. The first argument is a predicate on an array that is either evaluates to true, returning the second argument at the corresponding index in the array or false, returning the third argument instead. Now see how you get on re-implementing the code to do the prediction for the two-input Boolean AND function, as above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.1  -0.03 -0.06  0.01]\n",
      "Activations:\n",
      " [-0.1  -0.03 -0.06  0.01]\n",
      "Predictions:\n",
      " [0 0 0 1]\n",
      "Misclassifications\n",
      " [0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# Data set with class values in last column\n",
    "# dataset for AND function\n",
    "X=dataset[:,0:2]\n",
    "y = np.array([0,0,0,1])\n",
    "# Note: the bias weight is now the last!\n",
    "# fill in your weights here by assigning a weight vector to w\n",
    "w=np.array([0.04,0.07,0.1])\n",
    "# Add the values for the bias weights (-1) to the data matrix\n",
    "nData = np.shape(X)[0]\n",
    "X = np.concatenate((X,-np.ones((nData,1))),axis=1)\n",
    "# get the value of the activation function\n",
    "h = np.dot(X,w)\n",
    "print(h)\n",
    "yhat = np.where(h>0,1,0)\n",
    "err = yhat-y\n",
    "\n",
    "print('Activations:\\n', h)\n",
    "print('Predictions:\\n', yhat)\n",
    "print('Misclassifications\\n', err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding in weight updates to make the learning work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have spent some time just getting the weights and data in the right vector-matrix format to be able to do the prediction. What else do we need to get this thing to learn ?\n",
    "\n",
    "One thing we will need is some random initialisation for the weight vector. What sort of values would be appropriate for this initialisation?\n",
    "\n",
    "The initialisation will be done using a NumPy built-in. Note that we need weights for each of the inputs \"nIn\", plus one for the bias. Also, the \"nOut\" parameter is just a placeholder in case you want your Perceptron to predict more that one output at a time. Here we will just use one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.02752961]\n",
      " [-0.03397649]\n",
      " [ 0.04004982]]\n"
     ]
    }
   ],
   "source": [
    "nIn = 2    # still working with 2-input Boolean functions\n",
    "nOut = 1   # so a true/false classification output\n",
    "w = np.random.rand(nIn+1,nOut)*0.1-0.05 # Check: does this return a column vector?\n",
    "print(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other main thing we need is to see how the Perceptron training rule is implemented to update the weights for each attribute given all the information in the data matrix plus the misclassifications. Note that this implementation is a batch version, unlike the version in the lecture notes which is incremental. Both approaches have their place. Here we go for simplicity of implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What must the inner dimensions of the matrix multiplcation be for the weight update ? Check with the lecture notes to see what terms we will need. Recall that the augmented data matrix has $m+1$ columns, where $m$ is the number of inputs. However, the misclassifications, or errors, are of dimensionality $n$, because there is potentially one misclassification for every example in the dataset. What has to happen ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Correct: you need to transpose the augmented data matrix to ensure the inner dimensions match (they both must be of size $n$). Check you are sure before inspecting the code (it's just a one-liner). Here the parameter \"eta\" is the learning rate $\\eta$, which for this code is set to $0.25$. Once more $\\hat{y} - y$ will be our misclassification vector. Can you see why the updated weight vector $w$ has the values it does ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 1)\n",
      "(3, 4)\n",
      "(4,)\n",
      "[[-0.02752961]\n",
      " [-0.03397649]\n",
      " [ 0.04004982]]\n"
     ]
    }
   ],
   "source": [
    "eta=0.25\n",
    "print(w.shape)\n",
    "print(np.transpose(X).shape)\n",
    "print((yhat-y).shape)\n",
    "w -= eta*np.dot(np.transpose(X),(yhat-y).reshape((4,1)))# this is it - learning in one line of code!\n",
    "print(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can put it all together. Note that we need to set an upper limit for the number of iterations (T). Play with this code and run it as above for our Boolean functions. See what happens to the weights for \"OR\". Does the Perceptron learn this function? Now try \"AND\". Then try \"XOR\" (exclusive or). Now go back and experiment with the learning rate. Does anything change ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 0  Error: 0.5\n",
      "Iteration: 1  Error: 0.25\n",
      "Iteration: 2  Error: 0.0\n",
      "Iteration: 3  Error: 0.0\n",
      "Iteration: 4  Error: 0.0\n",
      "Iteration: 5  Error: 0.0\n",
      "Iteration: 6  Error: 0.0\n",
      "Iteration: 7  Error: 0.0\n",
      "Iteration: 8  Error: 0.0\n",
      "Iteration: 9  Error: 0.0\n",
      "Iteration: 10  Error: 0.0\n",
      "Iteration: 11  Error: 0.0\n",
      "Iteration: 12  Error: 0.0\n",
      "Iteration: 13  Error: 0.0\n",
      "Iteration: 14  Error: 0.0\n",
      "Iteration: 15  Error: 0.0\n",
      "Iteration: 16  Error: 0.0\n",
      "Iteration: 17  Error: 0.0\n",
      "Iteration: 18  Error: 0.0\n",
      "Iteration: 19  Error: 0.0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import division\n",
    "import numpy as np\n",
    "\n",
    "# Dataset with class values in last column\n",
    "dataset = np.array([[0,0,0],[0,1,1],[1,0,1],[1,1,1]])   # OR function\n",
    "# dataset for AND function\n",
    "#dataset = np.array([[0,0,0],[0,1,0],[1,0,0],[1,1,1]])   # OR function\n",
    "# dataset for XOR function\n",
    "#dataset = np.array([[0,0,0],[0,1,1],[1,0,1],[1,1,0]])   # OR function\n",
    "X = dataset[:,0:2]\n",
    "y = dataset[:,2:]\n",
    "nIn = np.shape(X)[1]    # no. of columns of data matrix\n",
    "nOut = np.shape(y)[1]   # no. of columns of class values -- just 1 here\n",
    "nData = np.shape(X)[0]  # no. of rows of data matrix\n",
    "w = np.random.rand(nIn+1,nOut)*0.1-0.05\n",
    "X = np.concatenate((X,-np.ones((nData,1))),axis=1)\n",
    "eta=0.25\n",
    "T=20\n",
    "# Train for T iterations\n",
    "for t in range(T):\n",
    "        # Predict outputs given current weights\n",
    "        h = np.dot(X,w)\n",
    "        yhat = np.where(h>0,1,0)\n",
    "        # Update weights for all incorrect classifications\n",
    "        w -= eta*np.dot(np.transpose(X),yhat-y)\n",
    "        # Output current performance\n",
    "        errors=yhat-y\n",
    "        perrors=((nData - np.sum(np.where(errors==0,1,0)))/nData)\n",
    "        # print(perrors, 'is Error on iteration:', t)\n",
    "        print('Iteration:', t, ' Error:', perrors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perceptron training on real data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, try this out on a real dataset, the standard diabetes dataset. You can download this from within your program. The rest of your program should work the same. Replace the lines defining the dataset, X and y variables with the code below. Perhaps surprisingly this simple algorithm actually learns to classify, but unfortunately, this basic implementation of neural learning is not likely to find a very good model. It's also not clear if it converges. You might want to increase the number of iterations from 20. Also, you could try transforming the data, for example, by making all attribute values lie in the same range. Search for methods of normalisation using the NumPy built-in functions \"np.mean()\" and \"np.var()\". For example, you could transform dataset ```X``` with this normalisation:\n",
    "```Z = (X - np.mean(X,axis = 0))/(np.var(X,axis = 0)**0.5)```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(768, 9)\n"
     ]
    }
   ],
   "source": [
    "import urllib\n",
    "# URL for a copy of the Pima Indians Diabetes dataset (UCI Machine Learning Repository)\n",
    "url = \"http://cse.unsw.edu.au/~mike/comp9417/data/uci_pima_indians_diabetes.csv\"\n",
    "# download the file\n",
    "raw_data = urllib.request.urlopen(url)\n",
    "# load the CSV file as a numpy matrix\n",
    "dataset = np.loadtxt(raw_data, delimiter=\",\")\n",
    "print(dataset.shape) # 8 attributes, 1 class, 768 examples\n",
    "X = dataset[:,0:8]\n",
    "y = dataset[:,8:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the full code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(768, 9)\n",
      "Iteration: 0  Error: 0.6510416666666666\n",
      "Iteration: 1  Error: 0.3489583333333333\n",
      "Iteration: 2  Error: 0.3489583333333333\n",
      "Iteration: 3  Error: 0.6510416666666666\n",
      "Iteration: 4  Error: 0.3489583333333333\n",
      "Iteration: 5  Error: 0.4908854166666667\n",
      "Iteration: 6  Error: 0.3489583333333333\n",
      "Iteration: 7  Error: 0.6510416666666666\n",
      "Iteration: 8  Error: 0.3489583333333333\n",
      "Iteration: 9  Error: 0.5572916666666666\n",
      "Iteration: 10  Error: 0.3489583333333333\n",
      "Iteration: 11  Error: 0.38671875\n",
      "Iteration: 12  Error: 0.3528645833333333\n",
      "Iteration: 13  Error: 0.6510416666666666\n",
      "Iteration: 14  Error: 0.3489583333333333\n",
      "Iteration: 15  Error: 0.6497395833333334\n",
      "Iteration: 16  Error: 0.3489583333333333\n",
      "Iteration: 17  Error: 0.3528645833333333\n",
      "Iteration: 18  Error: 0.6497395833333334\n",
      "Iteration: 19  Error: 0.3489583333333333\n",
      "Iteration: 20  Error: 0.58984375\n",
      "Iteration: 21  Error: 0.3489583333333333\n",
      "Iteration: 22  Error: 0.3385416666666667\n",
      "Iteration: 23  Error: 0.6497395833333334\n",
      "Iteration: 24  Error: 0.3489583333333333\n",
      "Iteration: 25  Error: 0.3606770833333333\n",
      "Iteration: 26  Error: 0.3528645833333333\n",
      "Iteration: 27  Error: 0.6497395833333334\n",
      "Iteration: 28  Error: 0.3489583333333333\n",
      "Iteration: 29  Error: 0.6223958333333334\n",
      "Iteration: 30  Error: 0.3489583333333333\n",
      "Iteration: 31  Error: 0.3528645833333333\n",
      "Iteration: 32  Error: 0.6497395833333334\n",
      "Iteration: 33  Error: 0.3489583333333333\n",
      "Iteration: 34  Error: 0.5716145833333334\n",
      "Iteration: 35  Error: 0.3489583333333333\n",
      "Iteration: 36  Error: 0.3723958333333333\n",
      "Iteration: 37  Error: 0.3489583333333333\n",
      "Iteration: 38  Error: 0.6497395833333334\n",
      "Iteration: 39  Error: 0.3489583333333333\n",
      "Iteration: 40  Error: 0.58203125\n",
      "Iteration: 41  Error: 0.3489583333333333\n",
      "Iteration: 42  Error: 0.3229166666666667\n",
      "Iteration: 43  Error: 0.6419270833333334\n",
      "Iteration: 44  Error: 0.3489583333333333\n",
      "Iteration: 45  Error: 0.3268229166666667\n",
      "Iteration: 46  Error: 0.5143229166666666\n",
      "Iteration: 47  Error: 0.3489583333333333\n",
      "Iteration: 48  Error: 0.4739583333333333\n",
      "Iteration: 49  Error: 0.3489583333333333\n",
      "Iteration: 50  Error: 0.58203125\n",
      "Iteration: 51  Error: 0.3489583333333333\n",
      "Iteration: 52  Error: 0.3372395833333333\n",
      "Iteration: 53  Error: 0.6497395833333334\n",
      "Iteration: 54  Error: 0.3489583333333333\n",
      "Iteration: 55  Error: 0.4986979166666667\n",
      "Iteration: 56  Error: 0.3489583333333333\n",
      "Iteration: 57  Error: 0.5130208333333334\n",
      "Iteration: 58  Error: 0.3489583333333333\n",
      "Iteration: 59  Error: 0.4127604166666667\n"
     ]
    }
   ],
   "source": [
    "from __future__ import division\n",
    "import numpy as np\n",
    "\n",
    "import urllib\n",
    "# URL for a copy of the Pima Indians Diabetes dataset (UCI Machine Learning Repository)\n",
    "url = \"http://cse.unsw.edu.au/~mike/comp9417/data/uci_pima_indians_diabetes.csv\"\n",
    "# download the file\n",
    "raw_data = urllib.request.urlopen(url)\n",
    "# load the CSV file as a numpy matrix\n",
    "dataset = np.loadtxt(raw_data, delimiter=\",\")\n",
    "print(dataset.shape) \n",
    "X = dataset[:,0:8]\n",
    "\n",
    "y = dataset[:,8:9]\n",
    "\n",
    "nIn = np.shape(X)[1]    # no. of columns of data matrix\n",
    "nOut = np.shape(y)[1]   # no. of columns of class values -- just 1 here\n",
    "nData = np.shape(X)[0]  # no. of rows of data matrix\n",
    "w = np.random.rand(nIn+1,nOut)*0.1-0.05\n",
    "X = np.concatenate((X,-np.ones((nData,1))),axis=1)\n",
    "eta=0.25\n",
    "T=60\n",
    "# Train for T iterations\n",
    "for t in range(T):\n",
    "        # Predict outputs given current weights\n",
    "        h = np.dot(X,w)\n",
    "        yhat = np.where(h>0,1,0)\n",
    "        # Update weights for all incorrect classifications\n",
    "        w -= eta*np.dot(np.transpose(X),yhat-y)\n",
    "        # Output current performance\n",
    "        errors=yhat-y\n",
    "        perrors=((nData - np.sum(np.where(errors==0,1,0)))/nData)\n",
    "        # print(perrors, 'is Error on iteration:', t)\n",
    "        print('Iteration:', t, ' Error:', perrors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, this code is just Python code so you can use this for further experimentation. Of course, it can be improved a lot!  A good starting point would be the NumPy documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2) Implementing a Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although real-world applications of neural networks are typically based on one of the many special-purpose libraries (such as TensorFlow, PyTorch, CNTK, etc.) it is possible and instructive to implement at least a basic neural network just using standard Python libraries. We start by implementing some key functions and concepts for a multi-layer neural network. Before coding the fully connected multi-layer neural network, let us code some basic functions needed for  the multi-layer neural network. We will need several libraries later so it is easiest to import them first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "from functools import partial\n",
    "import math, random\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  What is the sigmoid function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    # To do\n",
    "    return 1/(1+np.e(-x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the derivative of the sigmoid function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_der(x):\n",
    "    # To do\n",
    "    return sigmoid(x) * (1-sigmoid(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the output function for neurons?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neuron_output(w,x,b):\n",
    "    # To do\n",
    "    wx=np.dot(x, w)\n",
    "    return sigmoid( wx + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the softmax function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(y):\n",
    "    # To do\n",
    "    return "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to initialise the network weights?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initial_weight(input_dim,output_dim,hid_layers):\n",
    "    number_NN = hid_layers+[output_dim]\n",
    "\n",
    "    last_neural_number = input_dim\n",
    "    weight_list,bias_list = [],[]\n",
    "\n",
    "    for current_neural_number in number_NN:\n",
    "        # To do: code up some method to initialize weights and uncomment the following 2 lines \n",
    "        current_weights = np.random.randn(last_neural_number, size_l-1)\n",
    "        current_bias = np.random.randn(1, size_l-1)\n",
    "\n",
    "        last_neural_number = current_neural_number\n",
    "\n",
    "        weight_list.append(current_weights)\n",
    "        bias_list.append(current_bias)\n",
    "\n",
    "    return weight_list,bias_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many functions did you manage to implement?\n",
    "\n",
    "You should have been able to think of code for most of these functions from your knowledge of neural networks. If you did manage to get some code, great: below there is a reference implementation of a multilayer perceptron in which most of your functions should work if you added them.\n",
    "\n",
    "To test this implementation we will use a toy dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example application: simplified hand-written digit classification\n",
    "\n",
    "We will use a dataset of simplified \"hand-written\" digits for classification into one of ten classes (0-9). The representation is in a text format (see below) to make it easy to handle. \n",
    "\n",
    "For this dataset the inputs will be a 5x5 matrix of binary \"pixels\" (0 or 1, represented pictorially as '.' or '1' for input and '.' or '@' for output).\n",
    "\n",
    "The network structure will be:\n",
    "\n",
    "25 inputs (pixels)\n",
    "\n",
    "5 hidden units\n",
    "\n",
    "10 output units.\n",
    "\n",
    "The output unit with the largest value will taken as the predicted digit.\n",
    "\n",
    "We will run the network for 10000 iterations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the raw digit input and the target value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 1 1 1 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1]\n",
      " [0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0]\n",
      " [1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1]\n",
      " [1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1]\n",
      " [1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 0 0 0 0 1 0 0 0 0 1]\n",
      " [1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1]\n",
      " [1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1]\n",
      " [1 1 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1]\n",
      " [1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1]\n",
      " [1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 0 0 0 0 1 1 1 1 1 1]]\n"
     ]
    }
   ],
   "source": [
    "raw_digits = [\n",
    "      \"\"\"11111\n",
    "         1...1\n",
    "         1...1\n",
    "         1...1\n",
    "         11111\"\"\",\n",
    "\n",
    "      \"\"\"..1..\n",
    "         ..1..\n",
    "         ..1..\n",
    "         ..1..\n",
    "         ..1..\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         ....1\n",
    "         11111\n",
    "         1....\n",
    "         11111\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         ....1\n",
    "         11111\n",
    "         ....1\n",
    "         11111\"\"\",\n",
    "\n",
    "      \"\"\"1...1\n",
    "         1...1\n",
    "         11111\n",
    "         ....1\n",
    "         ....1\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         1....\n",
    "         11111\n",
    "         ....1\n",
    "         11111\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         1....\n",
    "         11111\n",
    "         1...1\n",
    "         11111\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         ....1\n",
    "         ....1\n",
    "         ....1\n",
    "         ....1\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         1...1\n",
    "         11111\n",
    "         1...1\n",
    "         11111\"\"\",\n",
    "\n",
    "      \"\"\"11111\n",
    "         1...1\n",
    "         11111\n",
    "         ....1\n",
    "         11111\"\"\"]\n",
    "\n",
    "def make_digit(raw_digit):\n",
    "    return [1 if c == '1' else 0\n",
    "            for row in raw_digit.split(\"\\n\")\n",
    "            for c in row.strip()]\n",
    "\n",
    "inputs = np.array(list(map(make_digit, raw_digits)))\n",
    "print(inputs)\n",
    "targets = np.eye(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation\n",
    "\n",
    "Here is a Neural Network object, providing the ability to define the learning rate, number of epochs/iterations, batch size, the number of layers and the number of neurons in each layer. The default setting of learning_rate, epochs, batch size and neural_numbers are 0.1, 1000, None, and \\[10\\] respectively. If batch_size is set to be None, that means all samples will be used for training in each iteration. \\[10\\] means that there is only one hidden layer with 10 neurons. If you want to change the number of hidden layers or the number of neurons, you can change the value of ```neural_numbers```. \n",
    "\n",
    "Compare your function code from above with the ones used in this implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetwork(object):\n",
    "    def __init__(self, learning_rate=0.1, epochs=1000, batch_size=None,neural_numbers=[10]):\n",
    "        self.learning_rate = learning_rate\n",
    "        self.epochs = epochs\n",
    "        self.batch_size = batch_size\n",
    "        self.neural_numbers=neural_numbers\n",
    "        self.layers=len(self.neural_numbers)+1\n",
    "        np.random.seed(77)\n",
    "\n",
    "    def fit(self,X,y):\n",
    "        self.X,self.y = X,y\n",
    "        self.initial_weight()\n",
    "        self.backpropagate(X,y)\n",
    "    \n",
    "    def forward(self,X):\n",
    "        output_list = []\n",
    "        input_x = X\n",
    "\n",
    "        for layer in range(self.layers):\n",
    "            cur_weight = self.weight_list[layer]\n",
    "            cur_bias = self.bias_list[layer]\n",
    "\n",
    "            # Calculate the output for current layer\n",
    "            output = self.neuron_output(cur_weight,input_x,cur_bias)\n",
    "            # The current output will be the input for the next layer.\n",
    "            input_x =  output\n",
    "\n",
    "            output_list.append(output)\n",
    "        return output_list\n",
    "\n",
    "    def backpropagate(self,train_x,train_y):\n",
    "        acc_list=[]\n",
    "        for iteration in range(self.epochs):\n",
    "            if self.batch_size:\n",
    "                n=train_x.shape[0]\n",
    "                # Sample batch_size number of sample for n samples\n",
    "                sample_index=np.random.choice(n, self.batch_size, replace=False)\n",
    "                x=train_x[sample_index,:]\n",
    "                y=train_y[sample_index,:]\n",
    "            else:\n",
    "                x=train_x\n",
    "                y=train_y\n",
    "\n",
    "            output_list=self.forward(x)\n",
    "            y_pred=output_list.pop()\n",
    "            # Record the accuracy every 5 iteration.\n",
    "            if iteration%5==0:\n",
    "                acc=self.accuracy(self.softmax(y),self.softmax(y_pred))\n",
    "                acc_list.append(acc)\n",
    "\n",
    "            loss_last=y-y_pred\n",
    "\n",
    "            output=y_pred\n",
    "\n",
    "            for layer in range(self.layers-1,-1,-1):\n",
    "                if layer!=0:\n",
    "                    input_last=output_list.pop()\n",
    "                else:\n",
    "                    input_last=x\n",
    "\n",
    "                if layer==self.layers-1:\n",
    "                    loss,dw,db=self.der_last_layer(loss_last,output,input_last)\n",
    "                else:\n",
    "                    weight=self.weight_list[layer+1]\n",
    "                    loss,dw,db=self.der_hidden_layer(loss_last,output,input_last,weight)\n",
    "\n",
    "                output=input_last\n",
    "                self.weight_list[layer] +=dw*self.learning_rate\n",
    "                self.bias_list[layer] +=db*self.learning_rate\n",
    "                loss_last=loss\n",
    "        self.acc_list=acc_list\n",
    "\n",
    "    def predict(self,X):\n",
    "        output_list = self.forward(X)\n",
    "        pred_y = self.softmax(output_list[-1])\n",
    "        return pred_y\n",
    "\n",
    "    def accuracy(self, pred, y_test):\n",
    "        assert len(pred) == len(y_test)\n",
    "        true_pred=np.where(pred==y_test)\n",
    "        if true_pred:\n",
    "            true_n = true_pred[0].shape[0]\n",
    "            return true_n/len(pred)\n",
    "        else:\n",
    "            return 0\n",
    "\n",
    "    def initial_weight(self):\n",
    "        if self.X is not None and self.y is not None:\n",
    "            x=self.X\n",
    "            y=self.y\n",
    "            input_dim = x.shape[1]\n",
    "            output_dim = y.shape[1]\n",
    "\n",
    "            number_NN = self.neural_numbers+[output_dim]\n",
    "            weight_list,bias_list = [],[]\n",
    "            last_neural_number = input_dim     \n",
    "            for cur_neural_number in number_NN:\n",
    "                \n",
    "                # The dimension of weight matrix is last neural number * current neural number\n",
    "                weights = np.random.randn(last_neural_number, cur_neural_number)\n",
    "                \n",
    "                # The number of dimension for bias is 1 and the number of current neural\n",
    "                bias = np.zeros((1, cur_neural_number))\n",
    "\n",
    "                last_neural_number=cur_neural_number\n",
    "\n",
    "                weight_list.append(weights)\n",
    "                bias_list.append(bias)\n",
    "                \n",
    "            \n",
    "\n",
    "            self.weight_list=weight_list\n",
    "            self.bias_list=bias_list\n",
    "\n",
    "    # Classical sigmoid activation functions are used in every layer in this network\n",
    "    def sigmoid(self, x):\n",
    "        return 1 / (1 + np.exp(-x))\n",
    "    \n",
    "    # Derivation of the sigmoid activation function\n",
    "    def sigmoid_der(self, x):\n",
    "        return (1 - x) * x\n",
    "\n",
    "    # Calculate the output for this layer\n",
    "    def neuron_output(self,w,x,b):\n",
    "        wx=np.dot(x, w)\n",
    "        return self.sigmoid( wx + b)\n",
    "\n",
    "    def der_last_layer(self,loss_last,output,input_x):\n",
    "        sigmoid_der=self.sigmoid_der(output)\n",
    "        loss = sigmoid_der*loss_last\n",
    "        dW = np.dot(input_x.T, loss)\n",
    "        db = np.sum(loss, axis=0, keepdims=True)\n",
    "        return loss,dW,db\n",
    "\n",
    "    def der_hidden_layer(self,loss_last,output,input_x,weight):\n",
    "        loss = self.sigmoid_der(output) * np.dot(loss_last,weight.T)\n",
    "        db = np.sum(loss, axis=0, keepdims=True)\n",
    "        dW = np.dot(input_x.T, loss)\n",
    "        return loss,dW,db\n",
    "\n",
    "    def softmax(self,y):\n",
    "        return np.argmax(y,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to run the implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n",
      "weight  2\n"
     ]
    }
   ],
   "source": [
    "Learning_rate=0.05\n",
    "nn=NeuralNetwork(learning_rate=Learning_rate)\n",
    "nn.fit(inputs,targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimenting with the implementation\n",
    "\n",
    "Parameter turning is not that easy in Neural Networks. To see this, let's investigate the relationship between learning rate and accuracy. Below is a function to test the effect of learning rate on accuracy. Run it and it should generate some plots to show the effect.\n",
    "\n",
    "If you want to try other values for the learning rate, or investigate the effect of other parameters, go ahead and change them and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvXmYXVWV9//5VmUeyFQQxkwY1CAKGAYHNCoi0EhwJmoDDk2j4sCr/WucEdRXoelWX20b0CgIMjnG9wdEGonYIkOIkTmQRCAhgSQ1JKmqpG4N6/1j75ucVO6turdyz51qfZ7nPPecfc7Ze51TZ9Xa41oyMxzHcRyn2miotACO4ziOkws3UI7jOE5V4gbKcRzHqUrcQDmO4zhViRsox3EcpypxA+U4juNUJW6gqhhJt0s6t9JyOE494XpVO7iByoGkZySdXGk5zOw0M7u21PlKWiCpT1K7pO2SVkn6UBH3XyLp+n2U4f2SnpXUIek3kqYOcO3Rkh6S1Bl/j+4nS3d8luw2Z19kc9LB9WrQ+8utV1dHGfsknbcv5aaFG6gKIWlEhUXYYGYTgP2Ai4BrJL20HAVLOhK4CvhHYDrQCfxnnmtHAb8FrgemANcCv43pWW42swmJbW2qD+BULa5XhelV5G/Ax4EV6Us3NNxAFYmkMyStlNQm6V5Jr0ycu1jSmlh7elzSOxLnzpP0Z0n/IakFuCSm/Y+kf5PUKunvkk5L3LNM0kcT9w907WxJ98Sy/1vSDwqpjVngNqAFSD7LdyWtk7QttlpOiumnAl8A3hdrin+L6ZMk/VjSRknPS/q6pMY8xX4A+J2Z3WNm7cCXgXdKmpjj2gXACOA7ZtZlZt8DBLx5sGdzagfXq7LrFWb2AzO7C9g52PNUCjdQRSDpWGAx8M/ANEJtZYmk0fGSNcBJwCTga8D1kg5KZHECsBY4APhGIm0V0ARcDvxYkvKIMNC1PwceiHJdQqhFFfJMDZLOjHmuTpx6EDgamBrzvlXSGDO7A/gmu1str4rXXwv0AC8BjgFOAbL/BGbEfzwz4rVHEmpvAJjZGiADHJFDxCOBh21Pn1wPx/Qsb5fUIukxSR8r5Lmd6sH1qiJ6VRuYmW/9NuAZ4OQc6T8ELuuXtgp4Y558VgIL4/55wHP9zp8HrE4cjwMMODAeLwM+Oti1wAzCRzwucf564Po8ci0A+oA2oAvoBT4zyDtpBV4V9y9J5k3oTugCxibSFgF358nrLuCCfmnPAwtyXPtl4KZ+aTcAl8T9ecDBQCPwWmAjsKjS35BvOf/urld731MRvep3zf8A51X6+8i1eQuqOGYCn421ljZJbcBhhH+QSDon0U3RBryCUIPKsi5Hni9kd8ysM+5OyFN+vmsPBloSafnKSrLBzCYT+sq/R78uM0mflfSEpK3xWSb1e5YkM4GRwMbEs19FqNHmoj2Wm2Q/YHux15rZ42a2wcx6zexe4LvAu/OU61Qnrle5SVOvagI3UMWxDviGmU1ObOPM7EZJM4FrgAuBafEjfZQwXpIlLdfxG4GpksYl0g4r5EYz6wL+FThK0lkAsV/8X4H3AlPis2xl97P0f451hJpeU+K97GdmR5Kbx4BsFwYKs+5GA0/lufaV/bpnXhnTcz4Se75zp/pxvYq39csmTb2qCdxA5WekpDGJbQRBUS6QdIIC4yX9QxyEHE/4wDYDKEwvfUU5BDWzZ4HlhAHiUZJeA7y9iPszwJXAV2LSRELXxmZghKSvsGfN7EVglqSGeP9G4PfAlZL2i/3vh0t6Y54ibyCMG50kaTxwKfArM8tV01tG6Cr5lKTRki6M6X8AkLRQ0pT49zge+BRh1p9TnbheVYdeEZ9pDMFAZv8uVWUTqkqYKuM2YEdiu8TMlgP/BHyf0He8mtCHjZk9TvgY/0L40I4C/lxGeT8AvAZoBr4O3EyofRXKYmCGpLcDS4HbCTWvZwmzfJJdG7fG32ZJ2Smq5wCjgMcJ7+YXwEGwazC3PTuYa2aPARcQFGoTQXE/ns1cYSHlF+K1GeCsmH8b8GHgrJgOcDbh77AduA74tqWwxsUpGa5XVaBXkd8T/gavBa6O+28o4tlSR3GQzKkzJN0MPGlmX620LI5TL7helRdvQdUJko6Lzf8GhTUVC4HfVFoux6llXK8qS6VXXTul40DgV4T1GuuBj5nZXysrkuPUPK5XFcS7+BzHcZyqxLv4HMdxnKqkbrr4mpqabNasWZUWw6lDHnrooS1mtn+l5SgXrktOGgxFj+rGQM2aNYvly5dXWgynDpH0bIXLXwycAWwys73WAMVFzN8FTid4sD7PzFbEc+cCX4qXfr2QKfiuS04aDEWPvIvPcaqfnwKnDnD+NGBu3M4n+LZDIRbQVwnOUI8HvippSqqSOk4JqZsWlOMMhXtXb+Fn9z3LG47Yn0XHzxj8hgpgZvdImjXAJQuB6yzMeLpP0mQFb98LgDvNrAVA0p0EQ3djuhI7A7FyXRtX37OGvr5KS5IOX/yHl3PY1HGDX1gAbqCcYc0ty9dx+6MvcOTB/X1s1hSHsKdHgvUxLV/6Xkg6n9D6YsaM6jTU9cJv/vo8Sx97kZfsn893bW2T6S2d5XUD5QxrWju7eeWhk7jwzXMrLcq+kMs5bj6nuTnXlZjZ1QR3N8yfP9/XnqTIlvYuZkwdx9KLqsqrUFXiY1DOsKa1M8OUcaMGv7C6Wc+eXrYPBTYMkO5UkJaODFPH1/w3VxbcQDnDmmCgRlZajH1lCXBO9AR+IrA1esJeCpwSvb1PIURjXVpJQZ1goKa5gSoI7+JzhjVtHd1MrvIWlKQbCRMemiStJ8zMGwlgZv9F8BB+OsELeCfwoXiuRdJlhDDjAJdmJ0w4lWNLe4ZjZkyutBg1gRsoZ9jS3dvH9q6equ/iM7NFg5w34BN5zi0mhHxwqoC+PqO1M8O08aMrLUpN4F18zrClrbMbgCnja76Lz6kRtu7oprfPmDahuitF1YK3oJxhS2tniHlY7V18TnWys7uXxX/+O51dvQXfs3VHqBT5JInCcAPlDBtaOjLc+MBzdMd1GhvbdgIw1Q2UMwQe+HsLl9+xigZB8DZVGBNGj+DlB9X0uruy4QbKGTYsWfk8VyxdtUfahNEjmLP/+ApJ5NQyW9pD5Pe7PruA2U3+DaWBGyhn2NDSkUGCp79+Go0Nu2u8xdR+HSdLc3voIvbxpPRIdZKEpFMlrZK0WtLFOc7/h6SVcXtKUlviXG/i3JI05XSGB62d3UwaO5IRjQ1I2rU5zlBo7sgwslFMHO31/LRI7c1KagR+ALyVsKL9QUlLzOzx7DVmdlHi+k8CxySy2GFmR6clnzP8qBOvEU6V0NzexbTxo72SkyJptqCOB1ab2VozywA3Ebwu52MR7mXZSZG2zm4m177XCKdKaO7IePdeyqRpoIrxpDwTmA38IZE8RtJySfdJOivPfefHa5Zv3ry5VHI7dYq3oJxS0uw+9VInTQNVsCdl4GzgF2aWXFAww8zmA+8HviPp8L0yM7vazOab2fz99x82EbmdIdLW2e0Gytlnunv7uOPRjTzfuoOmCe4RIk3SNFDFeFI+m37de2a2If6uBZax5/iU4xRNnTiGdSrMslWbueD6FWxp72LmtNIE5nNyM+gkCUkXAjeYWWuReT8IzJU0G3ieYITenyP/lwJTgL8k0qYAnWbWJakJeB1weZHlO3XMjkwv967ZQk9fYaGL+vqMzkwvUyrYJTNUXZJ0KvBdoBH4kZl9q9/5/wDeFA/HAQeY2eR4rhd4JJ57zszO3IdHcIAXtoUF3r/6+Gs5+lB3+pomhcziO5AwA28Fwenk0uicckDMrCcq5FKCYi02s8ckXQosN7Ps1PFFwE398nw5cJWkPkIr71vJ2X+Oc9ODz/G13xX/SRw6ZWwK0hRM0brks2Grj5a4/umoQybR0OAz+NJkUANlZl+S9GVCLJkPAd+XdAvwYzNbM8i9txFCASTTvtLv+JIc990LHDWo9M6w5cVtXYxsFL/5xOsKvmdUYwMvOaByYbaHqEu7ZsMCSMrOhs1nnRcRwnE4KdHc0cWksSMZ2ei+ttOmoHVQZmaSXgBeAHoIXXK/kHSnmf1/aQroOLlo68wwedwojjx4UqVFKYoh6FKu2bAn5Mp7oNmwsaxvmdlv8tx7PnA+wIwZM4p7qGGGTy8vH4WMQX0KOBfYAvwI+Bcz65bUADwNuIFyyk5LR+1NeBiiLpViNuwGSXOAP0h6JFdrzcyuBq4GmD9/fmEDe8OUsEDXDVQ5KKQF1QS808yeTSaaWZ+kM9IRy3EGJiy6rbl/EkPRpWJnw+4RuDA5G1bSMsL41IBd887ANLdnOHz/ynUVDycKMVC3AbvCREuaCMwzs/vN7InUJHOcAWjtzNSiF/Kh6JLPhq0Q61o6d8VvSrK5vYvjZk+tgETDj0IM1A+BYxPHHTnSHKestNbmotuidclnw1aGdS2dnHT53XnPHzxpTBmlGb4UYqCU/Ohjd4S773UqhpnR1pmp6JqmITIkXfLZsOVnXUsnAP/ytpcyt9/Mz8YGceKcaZUQa9hRiKFZGwd3fxiPPw6sTU8kJx8bt+6gu8fHrzu7e+jps5qbJIHrUs2wpSOsdTpl3nTmTp9YYWmGL4UYqAuA7wFfIsweuos4HdUpH79/7AXO/9lDlRajqth/Ys35QXNdqhFaYrRcdwZbWQrpXthEGJh1KsgzzR0A/O93HsUoXyDIqBENvHXe9EqLURSuS7VDc0eGBlGLM0XrikLWQY0BPgIcCewaGTSzD6col9OP1s5uRjSIs487zAOk1SiuS7VDNpRGo7syqiiFVMV/RvAh9jbgj4R1GNvTFMrZm6znBDdONY3rUo3Q0u6xnqqBQsagXmJm75G00MyulfRzwpRXp4y0dnTX4qQAZ09clwahp7evYA/1abIlhnN3KkshBiq7Uq1N0isIPsRmpSaRkxOPBlsXuC4NwKbtO3nTFcvoyPQOfnEZePurDq60CMOeQgzU1XFF+peAJcAE4MupSuXsRVtntwdHq31clwbg75s76Mj0suj4wzhsauW/9VPmHVhpEYY9Axqo6MRyWwywdg8wp5jMCwi0dh5wBcGFC8D3zexH8dy5BEUG+LqZXVtM2fVGa2eGow/z4Gi1yr7q0nCgJa49+scTZzHv4P0qLI1TDQw4ScLM+oALh5JxItDaacA8YJGkeTkuvdnMjo5b1jhNJcS0OYEQD+erseY5LAmeE7qZPN7HoGqVfdGl4UJ2cWyTh7JwIoV08d0p6XPAzQTfYQCYWUv+W4DiA60leRtwZ7YMSXcCpwI3FnBv2di6o5vTvnMP40eP4PZPn8SIuD7ph8vW8P0/PF2ycgzI9PYx1cegap2h6tKwIBuptgZdWDkpUYiByq7RSLrxNwbvoig00Nq7JL0BeAq4yMzW5bn3kP43VjrI2nPNnWzYuhMI3RMH7BeWtjzw92bGjR7BwhIOso5obGDh0Xu9Aqe2GKouDQs8Uq3Tn0I8ScweYt6FBFr7HXBjDAdwAXAt8OYC7614kLWWzswe+1kD1dLZzcsOnMiXzsjVo+kMV/ZBl4bFeK5HqnX6U4gniXNypZvZdYPcOmigNTNrThxeA3w7ce+CfvcuG0zWctOWMFCtHd17pM+sgllITnUxVF1KjOe+laAbD0pakiN0xs1mdmG/e7PjufMJlbyH4r2tQ3yM1Ghpz3ikWmcPCuniOy6xPwZ4C7ACGMxADRpoTdJBZrYxHp4JZIO2LQW+mZgYcQrw+QJkLSutHbsN1J7GqvbCkTtlYai6VHPjuc3tXZz7kwf4wfuPZX3rDr7460cGXYD74radvPllB6QpllNjFNLF98nksaRJBJctg91XSKC1T0k6E+ghRBo9L97bIukygpEDuLQaB5JbO7v32u/p7WPbzh53MunsxVB1iRocz/2/D2/k0ee3cc2f1jJp7Eiea+nkrGMGH0N996sP3eeynfphKIEHO4G5hVw4WKA1M/s8eVpGZrYYWDwE+cpGW2eGUSMayPT00RpbUNkQ0d6CcgqgUF2q6fHc5vYM0yaM5t/fe3Qps3WGAYWMQf2O3R90A2FN0y1pClUrtHZ2c9CkMby4beeuLr6sofKpsk5/9kGXam48t7u3b9d+c4ePLTlDo5AW1L8l9nuAZ81sfUry1BSt0cN4d0/fri6+7K/7zXNyMFRdqrnx3LZdXd5Gc3uXz85zhkQhBuo5YKOZ7QSQNFbSLDN7JlXJqoDVm7bz5d88RiZRG0zy5MZtHDd7Kt09ffz3Ey/yrh/em+jic4V09mJIulSL47nNHSEibUtHhpaODK+c4m66nOIpxEDdCrw2cdwb047LfXn98OfVzfxlbTMnzJ6ac/HgMTOm8O5XH0pbZzd3PPoCAGNHNvLSAycyd/qEcovrVD9D1qVaG89tjl4hWjoyu4L/OU6xFGKgRpjZrjnUZpaRNCy+tux40g0fPWGXG6N8fPDEmeUQyaltho0uNcclGBu37mT7zh73r+cMiUIM1GZJZ8ZuBCQtBLakK1Z10NbZzcQxIwY1To5TIHWtS3c98SLX3/csELq/AZ5v2wHAVA/+5wyBQgzUBcANkr4fj9cDOVfE1xseJNApMXWtSzc/uI771rYwd/oEXnLABI48ZBKPbdjGqEZxwpyplRbPqUEKWai7BjhR0gRAZrY9fbGqg9ZOD7PulI5616WWjgzHzJjMz//pxEqL4tQJg/ZdSfqmpMlm1m5m2yVNkfT1cghXadriNHLHKQX1rks+GcIpNYUMrpxmZm3Zg+hk8vT0RKoeQheft6CcklHXutTc3kXTBB9rckpHIQaqUdKur07SWGBYfIWtHd3egnJKSd3qUqYn+KD0FpRTSgqZJHE9cJekn8TjDxH8fNUNT2zcxs/vfw5LuCgzg/auHp8k4ZSSutWl7JIM9xjhlJJCJklcLulh4GSC48k7gLpa9HPjA89x/f3P7hVSffp+o3n1zCl57nKc4qhnXcouzHWfe04pKdSb+QtAH/Be4O/AL1OTqAK0dGSYNW08d39uQaVFceqfutKlxf/zdzZu3cGGrTsBmOZjUE4JyWugJB1BcEq5CGgGbiZMjX1ToZkXEKb6fwEfJfgP2wx82Myejed6gUfipc+Z2ZmFllsswemrT4Zw0qEUulSNbNq+k0v/7+OMamxgRKM4aNIY5jSNr7RYTh0xUAvqSeBPwNvNbDWApIsKzbjAMNV/BeabWaekjwGXA++L53aYWVkCyLR2dHPgpDHlKMoZnuyrLlVlRW/L9tCt992zj+a0ow4qVbaOs4uBZvG9i9AdcbekayS9hdzBz/KxK0x19D+WDVO9CzO728w64+F9hFg1ZafNW1BOugxZlxIVvdMI8aMWSZrX77JsRe+VwC8IFb0sO8zs6LiVtBci67Hcu/WctMhroMzs12b2PuBlhABnFwHTJf1Q0ikF5F1QqOkEHwFuTxyPkbRc0n2Szsp1g6Tz4zXLN2/eXIBIuWnt7N5rgoTjlIp91KWqrei1RIewPrXcSYtB10GZWYeZ3WBmZxA+/JXAxQXkXVCoaQBJHwTmA1ckkmeY2XxCYLbvSDo8h2xXm9l8M5u///77FyDS3uzs7mVHd69HwHVSZ4i6lHpFD4ZW2dsSZ+65p3InLYpy021mLWZ2lZm9uYDLBw1TDSDpZOCLwJlm1pUoa0P8XUuodR5TjKyFko386V18TjkpQpdSr+hFeYqu7LV0dNHYIPYb47rjpEOacSR2hamOMW/OBpYkL5B0DHAVwThtSqRPya64l9QEvA5ITq4oGb9/PAQa9AW5TpVStRW95vbge6+hoZihaccpnNQMlJn1ANkw1U8At2TDVMfQ1BBqehOAWyWtlJQ1YC8Hlkv6G3A38K1+s/9Kxjf+/ycAmDXNp8c6VUnVVvSaOzK+MNdJlUIX6g6JAsJUn5znvnuBo9KUDaC7t4+unj4++vrZzDt4v7SLc5yiMbMeSdmKXiOwOFvRA5bH4IfJih7snk7+cuAqSX2EymhJK3rN7V3u2shJlVQNVLWTHX+aMW1chSVxnPxUa0WvpSPDUVMmp5W946Q6BlX1tEUHl+6x3HGKp7ndu/icdBnWBqo1tqB8DZTjFEdXTy/bu3p8irmTKsPcQGVbUD5N1nGKYfciXfci4aTHsDZQ2S4+X6TrOMWxK7yGt6CcFBnWBqqlI3TxeVh3xymO5g6P/+SkT93P4nth606eb9uR89zTL25n1IgGxo5sLLNUjlN7PPr8Vrp6+gD463OtgDuKddKl7g3UL1es54qlq/KenzltHHHtiOM4A3DB9Q+xvnV3ZW9Eg9h/ohsoJz3q3kC9/ZUH84pDJuU9P9s9SDhOQVz5nlexM7agAA6YOJoJo+v+X4hTQer+65oxbZwvxHWcEnDCnGmVFsEZZgzrSRKO4zhO9SKznJ77aw5Jm4Fn85xuAraUUZyBqBZZXI69ySfLTDMbWsCxGqRGdMnl2JtqkaVkelQ3BmogJC2PMXEqTrXI4nLsTTXJUq1UyztyOfamWmQppRzexec4juNUJW6gHMdxnKpkuBioqystQIJqkcXl2JtqkqVaqZZ35HLsTbXIUjI5hsUYlOM4jlN7DJcWlOM4jlNjuIFyHMdxqpK6N1CSTpW0StJqSRenXNZhku6W9ISkxyR9OqZfIul5SSvjdnrins9H2VZJelsJZXlG0iOxvOUxbaqkOyU9HX+nxHRJ+l6U42FJx5ZQjpcmnnulpG2SPlOOdyJpsaRNkh5NpBX9DiSdG69/WtK5Q5WnlhmuehTzrrguVVKPYl6V0SUzq9sNaATWAHOAUcDfgHkplncQcGzcnwg8BcwDLgE+l+P6eVGm0cDsKGtjiWR5Bmjql3Y5cHHcvxj4dtw/HbgdEHAicH+Kf48XgJnleCfAG4BjgUeH+g6AqcDa+Dsl7k+p9Lddzm0461HMv6p0qdx6FPOriC7VewvqeGC1ma01swxwE7AwrcLMbKOZrYj724EngEMGuGUhcJOZdZnZ34HVUea0WAhcG/evBc5KpF9ngfuAyZIOSqH8twBrzCyfl4KsLCV5J2Z2D9CSI/9i3sHbgDvNrMXMWoE7gVOHIk8N43qUu8xK6VJZ9Qgqp0v1bqAOAdYljtcz8IdeMiTNAo4B7o9JF8bm7uJsUzhl+Qz4vaSHJJ0f06ab2UYI/wSAA8ogR5KzgRsTx+V+J1D8O6jYN1RFDGc9gurTpWrQIyiDLtW7gcoV6Cn1efWSJgC/BD5jZtuAHwKHA0cDG4EryyDf68zsWOA04BOS3jCQyCnKEQqQRgFnArfGpEq8k4HIV26l5KkmhrMeQRXpUg3o0UBlFy1TvRuo9cBhieNDgQ1pFihpJEGpbjCzXwGY2Ytm1mtmfcA17G5qpyafmW2Iv5uAX8cyX8x2N8TfTWnLkeA0YIWZvRjlKvs7iRT7Dsr+DVUhw1aPYrnVpEvVokdQBl2qdwP1IDBX0uxY8zgbWJJWYZIE/Bh4wsz+PZGe7IN+B5CdCbMEOFvSaEmzgbnAAyWQY7ykidl94JRY5hIgO3PmXOC3CTnOibNvTgS2ZpvuJWQRiW6Jcr+TBMW+g6XAKZKmxO6TU2LacGJY6lEss9p0qVr0KFtGuro01FkdtbIRZpQ8RZjF8sWUy3o9ocn6MLAybqcDPwMeielLgIMS93wxyrYKOK1EcswhzOD5G/BY9rmBacBdwNPxd2pMF/CDKMcjwPwSv5dxQDMwKZGW+jshKPJGoJtQe/vIUN4B8GHCIPNq4EOV/qYrsQ1HPYr5Vo0uVUqPYl4V0SV3deQ4juNUJfXexec4juPUKG6gHMdxnKrEDZTjOI5TlbiBchzHcaoSN1CO4zhOVeIGqggkmaQrE8efk3RJifL+qaR3lyKvQcp5j4KX6Lv7pR8s6Rdx/+ikV+QSlDlZ0sdzleUMP1yPhlzmsNMjN1DF0QW8U1JTpQVJIqmxiMs/AnzczN6UTDSzDWaWVeyjCetOipFhxACnJwO7FKtfWc7ww/UovwyuRwncQBVHD3A1cFH/E/1rbpLa4+8CSX+UdIukpyR9S9IHJD2gEGPm8EQ2J0v6U7zujHh/o6QrJD2o4BDynxP53i3p54TFcP3lWRTzf1TSt2PaVwiLIP9L0hX9rp8Vrx0FXAq8TyG+zPviavrFUYa/SloY7zlP0q2SfkdwpjlB0l2SVsSysx6vvwUcHvO7IltWzGOMpJ/E6/8q6U2JvH8l6Q6F2DGXJ97HT6Osj0ja62/hVD2uR65HhZH2CvR62oB2YD9CfJhJwOeAS+K5nwLvTl4bfxcAbYQYN6OB54GvxXOfBr6TuP8OQqVhLmG19hjgfOBL8ZrRwHJCfJcFQAcwO4ecBwPPAfsDI4A/AGfFc8vIsbodmEWM9QKcB3w/ce6bwAfj/mSCR4Hx8br17F5BPgLYL+43EVaLK5l3jrI+C/wk7r8syj0m5r02vucxwLMEP16vJrjsz+Y1udLfhW+uR4l7XI9KuHkLqkgseFW+DvhUEbc9aCHGTRfB/cfvY/ojhI8syy1m1mdmTxM+qpcR/FWdI2klIeTANILiATxgIdZLf44DlpnZZjPrAW4gBBwbKqcAF0cZlhE+9Bnx3J1mlo0TI+Cbkh4G/pvgSn/6IHm/nuCuBTN7kqBAR8Rzd5nZVjPbCTxOCNC2Fpgj6f9IOhXYtg/P5VQI1yPXo0IYqL/Tyc93gBXATxJpPcQuU0kiRB7N0pXY70sc97Hn36C/36msi/pPmtkeThUlLSDU/HKRy639viDgXWa2qp8MJ/ST4QOE2uarzaxb0jMEJRws73wk31svMMLMWiW9ihD87BPAewn+vZzaw/UI16OB8BbUEIg1nVsIA6VZniE0myFElBw5hKzfI6kh9qfPITh5XAp8TCH8AJKOUPCqPBD3A2+U1KQw8LsI+GMRcmwnhNrOshT4ZPyHgaRj8tw3CdgUlepNhJparvyS3ENQSCQdQahRrspzLQoD6w1m9kvgy4Qw1E4N4nrkejQYbqCGzpWE/uEs1xA+5geA/jWiQllFUIDbgQtik/xHhGb5ijggehWDtHwtuLb/PHA3wQvzCjP77UD39ONuYF52cBe4jPCP4uEow2V57rvB4L1oAAAfn0lEQVQBmC9pOUFZnozyNAN/jgOyV/S75z+BRkmPADcD58UunHwcAiyL3SQ/jc/p1C6uR3vjehRxb+aO4zhOVeItKMdxHKcqcQPlOI7jVCVuoBzHcZyqxA2U4ziOU5W4gXIcx3GqEjdQjuM4TlXiBspxHMepStxAOY7jOFWJGyjHcRynKnED5TiO41QlbqAcx3GcqsQNlOM4jlOVuIGqISTdLuncSsvhOPWI61f14QaqACQ9I+nkSsthZqeZ2bWlzlfSAkl9ktolbZe0StKHirj/EknX76MM75f0rKQOSb+RNHWAay1e1x63H+1L2U5lcf0a9P590i9JB0laImlD1J1ZQ82r3LiBqhIkVTq68QYzmwDsB1wEXCPppeUoWNKRhPg8/0gIbd1JiG8zEK8yswlx+2jaMjq1zXDWL0LE4TuAd5WpvJLhBmofkXRGDEjWJuleSa9MnLtY0ppYa3pc0jsS586T9GdJ/yGpBbgkpv2PpH+T1Crp75JOS9yzTNJHE/cPdO1sSffEsv9b0g8KqYVZ4DagBUg+y3clrZO0TdJDkk6K6acCXwDeF2uIf4vpkyT9WNJGSc9L+nqMSpqLDwC/M7N7zKydEOHznZLyRQ91hgmuX/uuX2b2opn9J/Bgoe+9WnADtQ9IOhZYDPwzMI3QClgiaXS8ZA1wEiGE89eA6yUdlMjiBGAtcADwjUTaKkKU0cuBH0shRHQOBrr258ADUa5LCK2TQp6pQdKZMc/ViVMPAkcDU2Pet0oaY2Z3AN8Ebo6tmVfF668FeoCXAMcApwBZ5Z8R/+HMiNceSYhYCoCZrQEywBEDiHqPpBck/aqWuiycwnH9Kpl+1S5m5tsgG/AMcHKO9B8Cl/VLWwW8MU8+K4GFcf884Ll+588DVieOxwEGHBiPlwEfHexaYAbh4x2XOH89cH0euRYQugHagC6gF/jMIO+kldDNBkFBr0+cmx7zGZtIWwTcnSevuwihuZNpzwML8lz/BmAUMBn4PvAoMKLS34lvQ9tcv3LeUzL9SlwzIj7DrEr/zQvdvAW1b8wEPhtrK22S2oDDgIMBJJ2T6J5oA15BqDllWZcjzxeyO2bWGXcn5Ck/37UHAy2JtHxlJdlgZpMJfeTfA96cPCnps5KekLQ1Psukfs+SZCYwEtiYeParCDXZXLTHcpPsB2zPdbGFrsCMmbUBnwZmAy8f5Pmc2sP1KzfF6lfNUumBw1pnHfANM/tG/xOSZgLXAG8B/mJmvZJWAsnuBEtJro3AVEnjEkp0WCE3mlmXpH8FVkk6y8x+E/vD/5XwLI+ZWZ+kVnY/S//nWEeo4TWZWU8BxT4GZLsukDQHGA08VYjMsfx83TRO7eL6FW/rl02x+lWzeAuqcEZKGpPYRhAU5AJJJygwXtI/KAzujyd8WJsBFKaVvqIcgprZs8BywsDwKEmvAd5exP0Z4ErgKzFpIqFLYzMwQtJX2LPF8yIwS1JDvH8j8HvgSkn7xX73wyW9MU+RNwBvl3SSpPHApcCvzGyvFpSkIyUdLalR0oQo5/PAE4U+n1OVuH6lp19IGkOo9AGMjsdVjxuowrkN2JHYLjGz5cA/EcZBWgmDnucBmNnjhI/wL4QP7Cjgz2WU9wPAa4Bm4OvAzYRaV6EsBmZIejuwFLid0KJ5FtjJnl0at8bfZkkr4v45hHGixwnv5hfAQbBrELc9O4hrZo8BFxAM1SaCwn48m7nCAsovxMPp8Vm2EQbAZwFnmFl3Ec/mVB+uXynpV2QHoSsd4Ml4XPUoDp45dY6km4EnzeyrlZbFceoN16908BZUnSLpuNjsb1BYS7EQ+E2l5XKcesD1qzz4JIn65UDgV4R1GuuBj5nZXysrkuPUDa5fZcC7+BzHcZyqxLv4HMdxnKoktS4+SYuBM4BNZrbX9M/oMuS7wOkE56DnmdmKeO5c4Evx0q9bAR6Gm5qabNasWSWS3nF289BDD20xs/0rVb7rklMPDEWP0hyD+ilheuh1ec6fBsyN2wkEtyYnKIRZ+Cown7DO4SFJS8ysdaDCZs2axfLly0skuuPsRtKzFRbhp7guOTXOUPQoNQNlZvdoYCeeC4HrLAyC3SdpsoKjxwXAnWbWAiDpTuBU4Ma0ZC2UTE8f7V11vXB72DJ6RAPjR1fnnKF61KXtO7vp7vXx73pkvzEjGNFYmtGjSmrkIey5GG19TMuXXnFO/e49rN3cUWkxnBT4xxNnctlZZXFEkAZVr0tbO7t51aW/55vvOIpZTeP4wI/ux+dn1Se3feok5h3c37Xm0KikgcrlOy2fT7Wcn7Kk84HzAWbMSNezfKanj7WbOzj55Qdw0tyKDUc4KfHSA2s69FTV69K61uCy7rq/PMOi42dgBv966ssYNypfiDCnVjlwUum8KFXSQK1nTweLhwIbYvqCfunLcmVgZlcDVwPMnz8/1fpYa2cGgAUvPYAPnjgzzaIcp1iqXpeSXePNHRkkOP8Nc2hscB+/Tn4qOc18CXBOdAJ5IrA1OkFcCpwiaYqkKYRAXEsrKCcAW9qDm62mCaMqLInj7EXV61Jze6jgmUFzexdTxo1y4+QMSprTzG8k1N6aJK0nzCYaCWBm/0VwDnk6wQFkJ/CheK5F0mXsDk98aXaQt5JkFWzahNGDXOk4paUedKm5Y7cf1ZaODNPGe0XPGZw0Z/EtGuS8AZ/Ic24xwdtv1dDSEQzUVFcsp8zUgy5lK3h9ZjR3ZFyPnIJwTxIFsquLb7y3oBynWLItqLYd3TS3d9HkPRFOAVTnwo8qpKUjw4gGsd9Yf2WOUyzZHojWjgyZnj5vQTkF4f9tC2DVC9v5z2VraJowiuBVxnGcfNzy4DouX/okZmH6/vlvmMNtj7wAQE+fsXVHN9N8spFTAG6gCuBv69sAOO+1syoriOPUAPeu2UJXTx+zm8Zz75pmjp89FYBvv+sonnxhO2bwjmOqYu29U+W4gSqA7ADvh18/u8KSOE7109yR4fD9J/DWedN5eP1Wtu3oYdSIBt53XLqL6Z36wydJFEBLRxdjRjYwbpTbc8cZjOb2MI18VPTH1rYjw9iR7jHCKR43UAUQFM5nHTlOITR3dDFtwihGjQj/XrZ2drtLI2dIuIEqgOaOjHuQcJwCMDNaOjJMHT96l4Fq7cww1g2UMwTcQBVAc0eXT4t1nALYtrOH7l6jaUKyi89bUM7QcANVAC3tGXdx5DgF0BwXtE8dv7uLb9uObsaN9PFbp3jcQA3CLx9az4atO913mOMUQHZB7rQJoxkdDVRbZzdjvAXlDIFBDZSkC6Mn5GHJL1esB+CUI6dXWBKn1hkOurRtZzcQoqpmW1A9fcY4n8XnDIFCWlAHAg9KukXSqSrClUK8fpWk1ZIuznH+PyStjNtTktoS53oT55YUWmapaenI8NZ503n1zKmVEsGpH4akS7WkR52ZXgDGjdptoMKxGyineAbtGDazL0n6MiGWzIeA70u6Bfixma3Jd5+kRuAHwFsJgdMelLTEzB5P5H1R4vpPAscksthhZkcX+0ClZkt7hmNmTK60GE4dMBRdqjU92rHLQDXS3rXbQPksPmcoFDQGFd35vxC3HmAK8AtJlw9w2/HAajNba2YZ4CZg4QDXLwJuLEjqMtHXZ7R2emgAp3QMQZdqSo92dAcDNWZkI6MadxslX6jrDIVCxqA+Jekh4HLgz8BRZvYx4NXAuwa49RBgXeJ4fUzLVcZMYDbwh0TyGEnLJd0n6aw8950fr1m+efPmwR6laLbu6Ka3z3yRrlMShqhLqetRvLckutSZaEF5F5+zrxQy97MJeKeZPZtMNLM+SWcMcF+u/nXLc+3ZwC/MrDeRNsPMNkiaA/xB0iP9u0HM7GrgaoD58+fny3vINO+akeQtKKckDEWXUtejKENJdClroMaObNw1iw9grLsJc4ZAIV18twG7wkRLmijpBAAze2KA+9YDhyWODwU25Ln2bPp1S5jZhvi7FljGnv3qZSG7psNbUE6JGIou1ZQe7ezuZczIBhoa5C0oZ58pxED9EGhPHHfEtMF4EJgrabakUQTl2WsWkaSXEvrh/5JImyJpdNxvAl4HPN7/3jS47ZGNfOHXj/CDu1d7C8opNUPRpZrSo85Mzy6nyqNG+CQJZ98opN2tOLAL7OqOKGT2X4+kC4GlQCOw2Mwek3QpsNzMskq2CLgpWQbwcuAqSX0EI/qt5KylNLny96tYs7kDgE+9ZS6AL9J1SkXRulRretSZ6d01ISJpoCaM9i4+p3gK+WrWSvoUu2t6HwfWFpK5md1G6NZIpn2l3/ElOe67FziqkDJKTXAMO5ot7V08/eJ2AKa4gXJKw5B0qZb0aEemd1drKeuLD7yS5wyNQrr4LgBeCzxP6A8/ATg/TaEqRU9vH22d3RwxfQIAT724nUljRzKy0T1COSWh7nVpR3fvrvGmPQyU+7J0hkAhXXWbCP3edU9LZxhzOmL6RO5d08yazR3M2X98haVy6oXhoEvJLr6Ght0TEL0F5QyFQQ2UpDHAR4AjgTHZdDP7cIpyVYSso8u5sQUF0OQz+JwSMRx0aUemN2fstEljR1ZAGqfWKaTv6mcEH2JvA/5ImOa6PU2hKkVzezBQs5vG0xhrf+5Fwikhda9LnZmenDP2kq0pxymUQgzUS8zsy0CHmV0L/AMVmsCQNtlp5QdMHM2UccEw+RRzp4TUvS7t7O5jrMd+ckpEIQaqO/62SXoFMAmYlZpEKdOZ6eGG+59lxXOte53bHWxtNFPGhS4J7zt3Skhd6VKSnd29/Pz+52jrzPiiXKdkFGKgro4xbL5EWCD4OPDtVKVKkaWPvcAXf/0oF/zsob3OtXRkaBBMHjuSlx44EYAj4q/jlIC60qUkf3hyE1/49SN0ZHqZ3bR7YtHU8aM4aW5TBSVzapkB2+KSGoBtZtYK3APMKYtUKbJpW2glZSdEJNnSHjyXNzSI7519DP/7nUcxcYwP7jr7Tj3qUpJN23YC8Md/WcDMabsN1Iovv7VSIjl1wIAtKDPrAy4skyxlITvO1NNnZHr69jjX0tG1y+9eQ4PcODklox51KUlz7H04bMq4Sovi1BGFdPHdKelzkg6TNDW7pS5ZSmyJ40wArZ17tqKa2zM+KcJJk7rSpSTJ3gfHKRWFTLfJrtH4RCLNqNEuimTXXnN7hun7jdnj3LyD96uEWM7woK50KUlLR5cvyXBKzqAtKDObnWMrSKEknSpplaTVki7Ocf48SZslrYzbRxPnzpX0dNzOLe6x8tPcnmF8nGXU3NG1x7kt7V00uUsWJyXqTZeSNLdnPCyNU3IK8SRxTq50M7tukPsagR8AbyX4HXtQ0pIc3pRvNrML+907FfgqMJ9Qw3wo3rv33PAiaenIMHf6RFaua9ujNZXp6WPbzh6vBTqpUW+6lKSlI8PLvffBKTGFdPEdl9gfA7wFWAEMqFTA8cDqGCgNSTcBCyksHs3bgDvNrCXeeydwKv2CsRWLmbGlvYvXvWQaK9e1sfyZ1l0GaeuOsETFx6CcFKkbXcry5Avb2Ly9i03buzjJK3dOiSnEWewnk8eSJhFctgzGIcC6xHHWe3N/3iXpDcBTwEVmti7PvYf0v1HS+URv0DNmzBhUoM5ML109fcxumsDE0SP42X3P8rP79oi+zaE+C8lJiXrSJYBtO7v5h+/9D719IQSV645Taobik6QTmFvAdbmm81i/498BN5pZl6QLgGuBNxd4L2Z2NXA1wPz58/c635+sr72mCaO4/TMn8cLWnXucHzOykSO9m8IpHzWrSxDWPvX2GRedfAQnHdHEUYdMKuQ2xymYQsagfsfuD7oBmAfcUkDe64HDEseHAhuSF5hZc+LwGnavql8PLOh377ICyhyQ7KSIpgmjOXTKOK/xOWWlnnQJwtRygPmzpnDsjCmlyNJx9qCQFtS/JfZ7gGfNbH0B9z0IzJU0mxCg7Wzg/ckLJB1kZhvj4ZnAE3F/KfDN6BYG4BTg8wWUOSDZFpSPMzkVom50CXYv2fCJRU5aFGKgngM2mtlOAEljJc0ys2cGusnMeiRdSFCQRmCxmT0m6VJguZktAT4l6UyCsrYA58V7WyRdRlBMgEuzg7z7giuUU2HqRpdgt3Nlr/A5aVGIgbqVEKY6S29MOy735bsxs9uA2/qlfSWx/3ny1ObMbDGwuAD5CmZL7OLz9RpOhagbXYLdbsOyoWkcp9QU4upohJntWjAU92vyi2xpD6EAcgVUc5wyUDe6BKHLfPK4kYxsLOTfiOMUTyFf1ubYdQCApIXAlvRESo/mDve151SUmtclM6O3L2wtHRnvLndSpZAuvguAGyR9Px6vB3KuiK92mjsyTPXuPady1Lwufev2J7nqnrW7jo+fVRe+bp0qpZCFumuAEyVNAGRm29MXKx2a27s4MOEc1nHKST3o0nV/2XNhu7egnDQZtItP0jclTTazdjPbLmmKpK+XQ7hS410STiWpB12aMGbPOq13mTtpUsgY1Glm1pY9iE4mT09PpHQwsxjvybv4nIpR87o0cXR/A+X65KRHIQaqUdKur1DSWKDmvsr2rh4yvX1M8xaUUzlqXpf2akG5PjkpUsgkieuBuyT9JB5/iODnq6ZwLxJOFVDzujTRu/icMlLIJInLJT0MnExwPHkHMDNtwUpNs3uRcCpMPejSuFF7/stwfXLSpNAVdi8AfcC7CDFsnhj48uoj65bFI+Y6FaamdclsT0fnrk9OmuRtQUk6guCUchHQDNxMmBr7pjLJVjJ2dvdy/s8eArxLwik/9aRLXT19exxPGjuyQpI4w4GBuvieBP4EvN3MVgNIuqgsUpWYTdtC6+mVh07ydVBOJagbXcoaqB+dM5+WzgzTXZ+cFBmoi+9dhO6IuyVdI+kt5A5+lhdJp0paJWm1pItznP9fkh6X9LCkuyTNTJzrlbQybkuKKbc/nd09AFzwxsORinoExykF+6RL1aJHAJmePk6a28TJ86bz3vmHDX6D4+wDeQ2Umf3azN4HvIwQ4OwiYLqkH0o6ZbCMJTUCPwBOIwRmWyRpXr/L/grMN7NXAr8ALk+c22FmR8ftTPaBHZleAHcS61SEfdGlatIjCAZq9Ah3DuuUh0G/NDPrMLMbzOwMQjTOlcBetbgcHA+sNrO10WvzTcDCfnnfbWad8fC+mH/J2WWgRrqBcirHEHWpavQIINPbxyg3UE6ZKOpLM7MWM7vKzN5cwOWHAOsSx+tjWj4+AtyeOB4jabmk+ySdlesGSefHa5Zv3rw5b8ad0UCN8xaUUyUUoUup6xEUrkuZnj5GeXgNp0wUslB3qOTqY7ccaUj6IDAfeGMieYaZbZA0B/iDpEeis83dmZldDVwNMH/+/Jx5A3R2u4FyapbU9QgK16Wunl5Gj3A9cspDmlWh9UByFPVQYEP/iySdDHwRONPMurLpZrYh/q4l9NsfM1RBdu4ag0rTHjtOKlSNHkFsQXkXn1Mm0vzSHgTmSpotaRRhHcges4gkHQNcRVCqTYn0KVmfZZKagNcBjw9VkM5MmMXnY1BODVI1egRuoJzyklqTwsx6JF0ILAUagcVm9pikS4HlZrYEuAKYANwap38/F2cavRy4SlIfwYh+y8yGbqC8i8+pUapJj8AnSTjlJdU+LzO7DbitX9pXEvsn57nvXuCoUsmxI9OLhE+PdWqSatGjvj6ju9d8koRTNobFl9aZ6WXsyEZfpOs4+0CmN3iR8BaUUy6GxZe2o7vXu/ccZx/JujnynginXAyLL21Hpte9SDjOPpJxA+WUmWHxpXVmehg30qeYO86+4F18TrkZFl9aZ6aXMd6Ccpx9ItuCcgPllIu6/9LueuJF/vT0Fsb5GijH2ScuWfIYAKMaXZec8lD3Bur5th0cNGkMJx3RVGlRHKem2b6zm8P3H8/LDppYaVGcYULdD8yc85pZnPOaWZUWw3Fqnl99/HWVFsEZZtR9C8pxHMepTdxAOY7jOFWJzPJ61q8pJG0Gns1zugnYUkZxBqJaZHE59iafLDPNbP9yC1MpakSXXI69qRZZSqZHdWOgBkLScjObX2k5oHpkcTn2pppkqVaq5R25HHtTLbKUUg7v4nMcx3GqEjdQjuM4TlUyXAzU1ZUWIEG1yOJy7E01yVKtVMs7cjn2plpkKZkcw2IMynEcx6k9hksLynEcx6kx3EA5juM4VUndGyhJp0paJWm1pItTLuswSXdLekLSY5I+HdMvkfS8pJVxOz1xz+ejbKskva2Esjwj6ZFY3vKYNlXSnZKejr9TYrokfS/K8bCkY0sox0sTz71S0jZJnynHO5G0WNImSY8m0op+B5LOjdc/LencocpTywxXPYp5V1yXKqlHMa/K6JKZ1e0GNAJrgDnAKOBvwLwUyzsIODbuTwSeAuYBlwCfy3H9vCjTaGB2lLWxRLI8AzT1S7scuDjuXwx8O+6fDtwOCDgRuD/Fv8cLwMxyvBPgDcCxwKNDfQfAVGBt/J0S96dU+tsu5zac9SjmX1W6VG49ivlVRJfqvQV1PLDazNaaWQa4CViYVmFmttHMVsT97cATwCED3LIQuMnMuszs78DqKHNaLASujfvXAmcl0q+zwH3AZEkHpVD+W4A1ZpbPS0FWlpK8EzO7B2jJkX8x7+BtwJ1m1mJmrcCdwKlDkaeGcT3KXWaldKmsegSV06V6N1CHAOsSx+sZ+EMvGZJmAccA98ekC2Nzd3G2KZyyfAb8XtJDks6PadPNbCOEfwLAAWWQI8nZwI2J43K/Eyj+HVTsG6oihrMeQfXpUjXoEZRBl+rdQClHWurz6iVNAH4JfMbMtgE/BA4HjgY2AleWQb7XmdmxwGnAJyS9YSCRU5QjFCCNAs4Ebo1JlXgnA5Gv3ErJU00MZz2CKtKlGtCjgcouWqZ6N1DrgcMSx4cCG9IsUNJIglLdYGa/AjCzF82s18z6gGvY3dROTT4z2xB/NwG/jmW+mO1uiL+b0pYjwWnACjN7McpV9ncSKfYdlP0bqkKGrR7FcqtJl6pFj6AMulTvBupBYK6k2bHmcTawJK3CJAn4MfCEmf17Ij3ZB/0OIDsTZglwtqTRkmYDc4EHSiDHeEkTs/vAKbHMJUB25sy5wG8TcpwTZ9+cCGzNNt1LyCIS3RLlficJin0HS4FTJE2J3SenxLThxLDUo1hmtelStehRtox0dWmoszpqZSPMKHmKMIvliymX9XpCk/VhYGXcTgd+BjwS05cAByXu+WKUbRVwWonkmEOYwfM34LHscwPTgLuAp+Pv1Jgu4AdRjkeA+SV+L+OAZmBSIi31d0JQ5I1AN6H29pGhvAPgw4RB5tXAhyr9TVdiG456FPOtGl2qlB7FvCqiS+7qyHEcx6lK6r2Lz3Ecx6lR3EA5juM4VYkbKMdxHKcqcQPlOI7jVCVuoBzHcZyqxA1UEUgySVcmjj8n6ZIS5f1TSe8uRV6DlPMeBS/Rd/dLP1jSL+L+0UmvyCUoc7Kkj+cqyxl+uB4Nucxhp0duoIqjC3inpKZKC5JEUmMRl38E+LiZvSmZaGYbzCyr2EcT1p0UI8OIAU5PBnYpVr+ynOGH61F+GVyPEriBKo4e4Grgov4n+tfcJLXH3wWS/ijpFklPSfqWpA9IekAhxszhiWxOlvSneN0Z8f5GSVdIelDBIeQ/J/K9W9LPCYvh+suzKOb/qKRvx7SvEBZB/pekK/pdPyteOwq4FHifQnyZ98XV9IujDH+VtDDec56kWyX9juBMc4KkuyStiGVnPV5/Czg85ndFtqyYxxhJP4nX/1XSmxJ5/0rSHQqxYy5PvI+fRlkfkbTX38KpelyPXI8KI+0V6PW0Ae3AfoT4MJOAzwGXxHM/Bd6dvDb+LgDaCDFuRgPPA1+L5z4NfCdx/x2ESsNcwmrtMcD5wJfiNaOB5YT4LguADmB2DjkPBp4D9gdGAH8AzornlpFjdTswixjrBTgP+H7i3DeBD8b9yQSPAuPjdevZvYJ8BLBf3G8irBZXMu8cZX0W+Encf1mUe0zMe218z2OAZwl+vF5NcNmfzWtypb8L31yPEve4HpVw8xZUkVjwqnwd8KkibnvQQoybLoL7j9/H9EcIH1mWW8ysz8yeJnxULyP4qzpH0kpCyIFpBMUDeMBCrJf+HAcsM7PNZtYD3EAIODZUTgEujjIsI3zoM+K5O80sGydGwDclPQz8N8GV/vRB8n49wV0LZvYkQYGOiOfuMrOtZrYTeJwQoG0tMEfS/5F0KrBtH57LqRCuR65HhTBQf6eTn+8AK4CfJNJ6iF2mkkSIPJqlK7HflzjuY8+/QX+/U1kX9Z80sz2cKkpaQKj55SKXW/t9QcC7zGxVPxlO6CfDBwi1zVebWbekZwhKOFje+Ui+t15ghJm1SnoVIfjZJ4D3Evx7ObWH6xGuRwPhLaghEGs6txAGSrM8Q2g2Q4goOXIIWb9HUkPsT59DcPK4FPiYQvgBJB2h4FV5IO4H3iipSWHgdxHwxyLk2E4ItZ1lKfDJ+A8DScfkuW8SsCkq1ZsINbVc+SW5h6CQSDqCUKNcledaFAbWG8zsl8CXCWGonRrE9cj1aDDcQA2dKwn9w1muIXzMDwD9a0SFsoqgALcDF8Qm+Y8IzfIVcUD0KgZp+Vpwbf954G6CF+YVZvbbge7px93AvOzgLnAZ4R/Fw1GGy/LcdwMwX9JygrI8GeVpBv4cB2Sv6HfPfwKNkh4BbgbOi104+TgEWBa7SX4an9OpXVyP9sb1KOLezB3HcZyqxFtQjuM4TlXiBspxHMepStxAOY7jOFWJGyjHcRynKnED5TiO41QlbqAcx3GcqsQNlOM4jlOV/D9qDDKWp1t54AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def test_LearnRate(Learning_rate,inputs,targets):\n",
    "    nn=NeuralNetwork(learning_rate=Learning_rate)\n",
    "    nn.fit(inputs,targets)\n",
    "    acc_array=np.array(nn.acc_list)\n",
    "    plt.plot(np.arange(acc_array.shape[0])*5,acc_array)\n",
    "    plt.title(\"Learning Rate:{}\".format(Learning_rate))\n",
    "    plt.ylabel(\"Accuracy\")\n",
    "    plt.xlabel(\"Number of iterations\")\n",
    "\n",
    "\n",
    "plt.figure()\n",
    "plt.subplot(2,2,1)\n",
    "Learning_rate=0.05\n",
    "test_LearnRate(Learning_rate,inputs,targets)\n",
    "\n",
    "plt.subplot(2,2,2)\n",
    "Learning_rate=0.1\n",
    "test_LearnRate(Learning_rate,inputs,targets)\n",
    "\n",
    "plt.subplot(2,2,3)\n",
    "Learning_rate=0.5\n",
    "test_LearnRate(Learning_rate,inputs,targets)\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "Learning_rate=1\n",
    "test_LearnRate(Learning_rate,inputs,targets)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
